[Perf][SM70] Add opt-in native-g32 AWQ QPN M1 dispatch - #521

Purpose

Connect the AWQ M=1 operator from #519 to a narrow, default-off runtime route for Qwen3.8 TP4 single-token decode, and record the full-model evidence for it.

Depends on #519 (the operator) and #520 (the layer contract helper and the grouped 2 to 8 token baseline both arms ran on). This branch contains both so it can be built and tested as a whole; the commits specific to this PR are the last two, 4d95867e3e (dispatch) and 356c1bafff (acceptance documentation). The rest of the diff disappears once #519 and #520 merge.

• Add VLLM_SM70_AWQ_QWEN38_QPN_M1, default 0, accepting only 0 or 1.
• At initialization, require the TP4 / E512 / native and prepared group-32 contract, the batched TurboMind banks, interleaved W13 and the legacy single-token compact admission. An explicit opt-in on an unsupported layer, or on a build without the native operator, fails closed.
• At execution, only a physical single token selects the route: contiguous FP16 (1, 2560) input, INT32 (1, 10) expert IDs, FP32 (1, 10) router weights. Every other physical batch shape, including padded CUDA Graph batches, keeps its existing route.
• Reuse the existing prepared banks and the existing intermediate and output buffers. No DSO loading, no compilation at startup, no additional weight copy, no change to grouped decode, prefill, router, shared experts, attention, graph policy or MTP.
• Rollback: set the flag to 0 and restart; changing the variable cannot replace an already captured graph.

Full-model performance

4x Tesla V100 PCIe 32 GB, TP4/MTP0, FP16 activations and KV, native-group-32 AWQ checkpoint, 4-byte metadata, frozen prompt token IDs, prefix cache off, 8192 max batched tokens, ignore_eos=false, 320 output tokens, one score per cell and arm. Both arms include #520's grouped-decode route and the same separately tracked QSA page4 fix (#494). Pure aggregate decode tok/s over the common all-active window:

C1 gains 18.0% in this pair; the same-day NVFP4 reference on the same contract was 60.26 tok/s, so the remaining gap is about 2.0%. An earlier prototype pair on the same contract measured 51.50 to 59.16 tok/s. C4 and C8 do not take this route except in a one-token drain tail and are unchanged within run-to-run noise.

Per-step GPU service time, averaged over ranks, from the traces of the prototype pair: W13 1.418 to 0.736 ms, W2 including weighted reduction 1.139 to 0.505 ms, the format-specific prepare kernel 0.988 ms to absent; QSA, HC and shared-expert gate unchanged. All 48 layers hit the new W13/W2 kernels; the old prepare and standalone reduce kernels are absent from every graph.

Memory after startup is identical between arms on all four ranks: 563 KV blocks, 5,762,813,952 KV bytes, 28,605,358,080 PyTorch allocated bytes per rank.

Numerical acceptance

Both paths approximate the same fixed AWQ-weight computation and neither is ground truth. Each was compared against the same independently decoded-weight FP64 reference; holding the checkpoint fixed separates this change's arithmetic perturbation from the quantization error present in both.

• 1,152 captured layer/rank/arm samples at three fixed prefixes: both implementations' W13 and W2 stay within the retained rounding bounds (maximum bound fraction QPN 0.174/0.966, legacy 0.174/0.954).
• Same-runtime 65-case quality pair, official scoring, natural EOS:

47/65 output token streams are identical. One GSM8K regression and one IFEval improvement remain and are not netted against each other; this is not a zero-regression or statistical non-inferiority result.

• The GSM8K divergence is attributable: at output index 120 the legacy logits for token IDs 4003 and 16526 are exactly tied at 21.59375, while the QPN path gives 21.625 and 21.578125, one to two FP16 steps apart, and the later tokens follow. The legacy path itself reproduced the failed answer at the same 256-token budget in a repeat, so the failure is not exclusive to the new kernel. At a captured 64K step, 13 of 48 layers change expert membership within their own router scores; the first change reverses a 0.0117 score margin, so global logit differences (maximum 1.75 in the focused capture, 5.05 in an earlier full trace) are larger than the local rounding error.
• The analysis tool was corrected to use argmax rather than the first index of topk(2) at ties; raw tensors and scores did not change.

Current evidence supports numerical trajectory variation without a systematic kernel defect on the tested inputs. It does not establish broad quality non-inferiority or production readiness, which is why the route is opt-in.

Duplicate-work check

Open-PR searches on 2026-09-06 in 1CatAI/1Cat-vLLM for "AWQ QPN", "QPN M1", "awq_qpn", "AWQ M1" and "Qwen3.8 AWQ decode" found no competing AWQ single-token route. #510 tunes the NVFP4 M=1 kernel; #509 is conversion-cache release; #504 is NVFP4 batched work. Base is main@755baae1d0.

Tests

• CPU on this branch: tests/quantization/test_awq_qpn_sm70.py, test_sm70_awq_active_grouped_decode.py, test_sm70_awq_indexed_prefill.py, test_sm70_awq_compact_metadata.py: 69 passed. Covers default-off never touching the native lookup, rejected flag values, unsupported layers failing closed, missing native build failing closed, both metadata layouts, no sidecar loading, single-physical-token admission across 0/1/2/4/5/8/128/8192 tokens, dtype and stride rejection, rollback, and the real MoE branch calling the native op with the existing banks.
• Pre-commit on the changed files, including mypy, and git diff --check: clean.
• The runtime Python files are the same commits that produced the GPU-validated build; the native core used for the full-model pair is e5083c1f...8916e1.

Provenance

Fork review draft: Leonccaa#12 (same commits, rebased onto #520 plus the same operator change as #519). Design and acceptance notes: docs/design/sm70_awq_qpn_m1.md.

AI assistance was used. OpenAI Codex implemented the dispatch, tests, attribution and evidence under Leon's direction; Claude Code reviewed the diff, re-ran the CPU tests and pre-commit on this branch, and prepared this submission. Human review is still required.

🤖 Generated with Claude Code